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ABSTRACT 

The increasing use of job training or retraining and 
employment programs in response to high rates of uncmplojnnent in 
France has spurred interest in program evaluation to see what results 
have been achieved. The contribution of traditional standardized 
evaluations and of emerging methodologies is being examined. 
Traditional evaluation methods focus on postprogram return to 
employment. The standard methodology consists of quantitative 
statistical surveys. Two very different approaches can be 
distinguished: the first targets the situation of program leavers at 
a given date, and the second traces the path of a cohort of potential 
public policy beneficiaries over time. Whether derived from 
transversal or longitudinal evaluation methods, such observations are 
often related to standard, classic sociodemographic variables. The 
rates of return to employment after participation in a program, as 
well as the description of individual progress, show that there is no 
determinist relationship between participation in the programs and 
return to employment. Thus, these evaluations suggest a greater 
complexity of entry and reentry processes, which other approaches 
attempt to examine. Two such approaches can be identified — one based 
on the strategies used by the beneficiaries and the other analyzing 
the interactions between the agents of entry (local representatives 
of national agencies, local employment agencies, training 
organizations, and potential or real employers) and specific 
socioeconomic conditions. The methods for identifying these variables 
consist of non- or semidirective interviews with the participants. 
Better qualitative evaluation methods, coupled with quantitative 
approaches, are needed. (KC) 

Reproductions supplied by EDRS are the best that can be made 

from the original document. ^» 
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Evaluation of Public 
Policies on Employment: 

Issues, Tools and Methods 



The employment situation in France has suffered sharp and 
continuing decline since the middle of the 1970s. This overall 
scarcity of employment has been accompanied^ however, by 
more specific problems: young people entering the labour 
market are particularly vulnerable, and long-term 
unemployment is excluding greater and greater portions of the 
workforce (women and low-skilled workers) from the job 
market. The 1980s saw the development of multiple large-scale 
public interventions aimed at remedying this situation. Because 
of their cost on the one hand and the persistence of 
uneriployment on the other, there has been growing interest in 
the evaluations of such policies. Predominantly statistical, these 
tools have become increasingly refined and diversified. 
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UNEMPLOYMENT AND PUBLIC 
POLICIES: THE ISSUES OF EVALUATION 

Uni:mployment has been growing at a steady rate for 
several years, generally around 10 percent; in 
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March 1 993* it hit the three-million mark. Various forms 
of selectivity have accompanied this overall scarcity of 
employment and have become increasingly pronounced. 
The phenomena are interrrelated, but two veiy marked 
trends should be singled out. Young people under 
twenty-five years of age are, relative to other 
economically active categories, the most vulnerable to 
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unemployment. Their unemployment rate remains well 
above that of the national average, despite a certain 
reduction in the differential over ttie past ten years 
(19.7 % versus 6.5 % In 1982; 19.1 % versus 10.2 % In 
December 1992. taking seasonal variations into 
account). The period of transition from school to work 
has become longer, particularly for those young people 
with the lowest degrees. It has also become more 
complicated, notabfy with the increase in precarious 
forms of employment and the implementation of 
successive programmes for post-secondary alternating 
training. Along with the rise of long-term unemployment 
(more than one year), the risks of occupational and 
social exclusion have increased. The rate of those 
unemployed for one year or more has gone from 12.8 
percent in December 1975 to 24.7 percent in December 
1982 and to 31.7 percent in December 1992. 

Young people and the long-term unemployed have thus 
become the pri\11eged targets of public policies. Over the 
past few years, nearly 800,000 young people and 
300,000 to 400,000 long-term unemployed have 
entered assistance programmes for job entry or re-entry 
each year. The rapid revision and the diversity of the 
programmes, which offer varying proportions of training 
courses, subsidies for hiring (jobs that are "subsidized" 
for fixed or Indefinite periods) and activities in the 
noncommercial sector, attest to both the significant 
difficulties In finding solutions to unemployment and a 
constant effort to improve the effectiveness of the 
measures. The scale of public interventions coupled 
with the persistence of unemplo3nment explain the 
gro\^1ng interest in evaluating the effects of the 
programmes on beneficiaries. 

In proposing a preliminary analysis of the tools and 
methods used for evaluating the effectiveness of these 
public programmes, this article addresses two main 
issues: the contribution of traditional standardised 
evaluations and that of newly emerging methodological 
trends. 



An Inventory of Methodologies 

In the context of its participation in the Technical Support Croup i 
set up by the Ministry of Labour's European Social Fund Mission^ t 
CEREQ undertook an initial exploration of thisjield through the I 

. ancdysisof sorne sixty evaluations carried oul in Frarvcebetu>een j 
1989 and 1992 and dealing with measures related to the t 

: European Social Fund's objectives 3 and 4 (unemployed young | 
people and long-term unemployed). This study followed a t 
collaboraV'^ii with the European Social Fund itself, within a ij 
working jroup that elaborated a European methodology for t 

: ex-post evaluaticn of jointly funded activities. [See Bibliography.] ;! 



A TRjWITIONAL EVALUATION: 
STATISTICAL DESCRIPTION Of 
BENEFICIARIES' EVOLUTION 

THESE evaluations focus on the post-programme 
return to employment. Hie standard methodology 
consists of quantitative statistical surveys. Two very 
different approaches can be distinguished: the first 
targets the situation of programme -leavers at a given 
date, while the second retraces the patli of a cohort of 
potential public-policy beneficiaries over time. 

ERJC 



I Measuring the Entry Rate 

The first option describes the evolution of beneficiaries 
of one or several measures on the basis of a single, 
selective observation. The methodological tool is 
generally a survey of the individuals involved, carried 
out three to six mon'Jis after they leave the programme 
to be evaluated. This is the oldest form of evaluation, 
and it is envisioned at the outset of every new measure. 
In France, it comes out of a tradition of counting and 
follow-up of beneficiaries that was set up by the 
administration and has since become a legal 
requirement. The government sees this as a means of 
guiding the activities undertaken on the basis of simple, 
homogeneous, standardised information. The results 
often come from the analysis of the individual follow-up 
sheets that the administration requires from training 
providers. The rate of return to emplo)rment, sometimes 
interpreted in terms of placement rate, is often 
considered the strategic indicator that sums up the 
effectiveness of the measures. 

This kind of evaluation encompasses a greater variety 
of situations or positions occupied at tlie date of the 
inquiry (unemployment, training course, inactivity, 
stable employment, precarious employment, subsidised 
employment, etc.). On this basis it is possible to make 
more detailed comparisons of the specific effects of each 
measure-traineeship, for example, generally seems to 
be less effective than sandwich training. Similarly, the 
evolution of those leaving a given programme can be 
studied in terms of their social characteristics: sex, age, 
qualification, training level, etc. The point, therefore, is 
to evaluate the degree to which public assistance 
policies for entr^^ and re-enti^^' compensate, reinforce or 
have no eflect on the mechanisms of selectivity at work 
on the labour market. 

These methods of intermittent observation have one 
limitation. The proportion of beneficiaries obtaining a 
job at the end of six months cannot be considered stable 
because the jobs obtained are often precarious. In 
addition, the results depend directly on the moment 
chosen to cany out the obser\'ation. Thus this approach 
is complemented by longitudinal analyses taking the 
time variable into account. 

I Measuring the Impact of Programmes 
on Individual Trajectories 

In France, longitudinal evaluation methods were first 
used to measure the effectiveness of the academic tracks 
of the initial training system. They have now been 
applied to the evaluation of public polici2s in the 
struggle against unemployment. These evaluations 
consist of a fairly long follow-up {3 to 5 years) of a cohort 
of individuals (young people or the long-term 
unemployed) involved in processes of mobility on the 
labour market. TTiey are thus massive and expensive to 
implement. They rely on retrospective interviews or on 
the panel approach (several interviews of the same 
people over time). Periods in the public employment 
programmes are thus seen as so many steps in career 
patterns, and tlielr effects are observed over time rather 
than instantaneously, at a precise date. 

These longitudinal approaches have allowed the 
idenUflcaUon cf typologies for entry paths. It has been 
possible to test whether the periods In the programmes 
(or their absence) significantly modify the career 
patterns. Th ese analyses provide a context lor 
interpreting the role of public measures in terms of 
direct effect or effectivf^ness per se. They emphasise that 
I the links between public policies and access to 
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employment cannot be reduced to a direct mechanical 
relationship but depend Instead on adjustments 
repeated over time, as the behaviours of the young 
people and the firms' choices are shaped and 
reorganised. 



Differences in Integrtttion Itinenaiem 

] An inquiry carried out by CEBEQ on a sample group of young 

^ people who left secondary education in 1986 studied their 

] career patterns over three and a half years. Three kinds of 

] evolutions were ideofi/ied, wtth entry programmes assuming 

■ different sigrdjkance and functions in each case: 

(1) A process of entry into working life that is stable or in the | 
process of becoming stable. Here, potential initial problems l 
were repolved by a moderate recourse to the programmes $ 
(50,9 % of young men and 60,2 % of young women). | 

(2) An evolution marked by an intensiife period in the ^ 
programmes. Such programmes are mainly oriented to I 
ovei coming the major obstacles to Job access (1 8.4 % of young | 
men and 22,3 % of young womer^. I 

(3) An evolution where unenrqAoyment is domtnanL There are | 
frequent periods tn programmss, but these neither resolve the f. 
intense difficulties of entry into working ii/e nor rule out the ^ 
danger of exclusi:>n (12,5% of young jnen and 17.6% of young 1 
women), | 

: 118,4 % of the young men were not included in the statistical I 
: data because they had been out of military service less than six ji 
. morvths at the time of the inquiry,] | 

In addition, the creation of the cohorts on the basis of 
criteria independent of the public policies (e.g., all the 
young people leaving school in a given year, or all the 
indi\1duals becoming unemployed in a given month) 
brings out certain effects of the programmes which had 
previously been overshadowed. It is possible to evaluate 
the proportion of individuals in a cohort affected by 
public policies, to measure the selectivity of the 
programmes by comparing profiles of beneficiaries and 
non-beneficiaries and to identify the phases of the 
evolutions corresponding to the periods in programmes, 
etc- 



A Panel Study \ 

For a number of years CEREQ has also been seeking to improve 
knowledge of the role and significance of public programmes of 
assistance for youth entry into working life relative to the • 
transition from school to work. These evaluations are carried ? 
out in theframeux:>rk of the National Observatory on Entry into ? 
Working Ufe (see Training and Employment no. 3), which i 
inifolves three operations: the decentralised entry survey; \ 
national inquiries on career paths, and specialized ] 
investigations Among the lasU the one that is most specificaRy ] 
aimed at the evaluation of the effect of measures is the { 
telephone panel on "public measures for the trcdning and | 
labour-market entry of young people", which was setup tnl 987. j- 
This permits foUowup on two cohorts of young people who left | 
secondary school before the baccalaw^at (tn 1986 and 1989) I 
and apprenticeship (tn 1989), A first telephone survey of the \ 
second cohort took place eighteen months after the interruption >. 
of studies. At present, three series of retrospective inquiries have i 
been conducted (in December 1990, 1991 and 1992), A fourth I 
inquiry is envisaged for December 1993 and will be '( 
accoinpanied by the survey of a representative sampling of 
various non-respondants from the previous series. The panel | 
study of this second cohort is part of a group of studies | 
sponsored by the Ministry of Labour wtth tlie participation of | 
various of CEREQ* s institutiorwl partners. It Jios served as the | 
base saniple for studies centred on one or another specific ? 
programmes and conducted by these partners (DARES, \ 
delegation on employmenU delegatior on vocational training), | 



Whether derived from transversal or longitudinal 
evaluation methods, observations are often related to 
standard, classic socio-demographlc variables. Yotmg 
people are above all characterised by their level of 
schooling (and thus their academic difiSculties), and the 
long-term unemployed by their level of qualification, 
their sex or their age (thus, factors of employability). 
These categorisations are based implicitly on a 
mechanistic conception of the role of public policies: the 
idea is to niake up for a deficiency or a well-identified 
ack that prevents or delays emplojnnent. For young 
people, the deficiencies are situated on the level of 
training; for the long-term unemployed, on the more 
undefined level of employability. In fact, the rates of 
return to employment after participation In a 
programme, as well as the description of individual 
evolutions, show that there is no determinist 
relationship between participation in the programmes 
and return to employment. Tlius these evaluations 
suggest a greater complexity of entiy and re-entry 
processes, and this is v^at other approaches attempt 
to examine. 



REFINEMENT OF EVALUATION 
APPROACHES: STRATEGIES AND 
LOCAL ECONOmC CONTEXT 

FOR several years various attempts have been made 
to diversify the nature of the explanatory variables. 
Two of them can be singled out: one based on the 
strategies of the beneficiaries and another analysing, on 
the loSl level, the institutional interactions between thv^ 
agents of entry (local representation of national 
ministries, local employment agency, training 
organisations and potential or real employers) and 
specific socio-economic conditions. 

The methods for identifying these variables are still 
rudimentary. They consist of non- or semi-directive 
interviews with the various interlocutors involved, often 
on a very local level. 

I Ben^ciaries* Strategies 

Certain evaluations analyse the representations (of self, 
situation, environment), behaviours and strategies of 
the Individuals involved and show that they react to the 
measures as players belonging to the social space of 
entry into working life. This approach leads to not 
limiting the evaluation to the direct "employment effects" 
that are usually considered the best indicators of 
success. The effects of participation in programmes on 
the beneficiaries' personal development are sometimes 
analysed as the most relevant dimension. A great 
Importance is attached to renewing the energy of the 
Individual, for example (but not necessarily), around the 
formulation and development of a career plan. Hie idea 
of the "travelled path" (development of autonomy, 
know-how, the ability to elaborate life projects, 
savoir-faire) thus becomes as important as entry into 
working life in the narrow sense. 

The evaluations analyse how the participation in public 
employment programmes falls subjectively and 
strategically within the framework of individual 
evolutions. They seek to understand, for example, the 
moti\^tions for accepting, and above all, for refusing, to 
enter public programmes. The underlying meaning of 
these motivations is sought in a life history, a specific 
sociological and economic context, in attitudes of 
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adaptation (opportunities seized) or of projection into 
the future, 

I The Contextual Approach: Locally-Based 
Programmes 

Taking into account interactions on the local level 
during the implementation of a programme and its 
operaUonal procedures brings out the potential brakes 
on the success of such implementation: the degree of 
motivation of one institutional partner or another 
(essenUally the employers in the case of the long-term 
unemployed), suspiciousness or rivalry among 
institutions, dependency of one player in relation to the 
others {more specifically the training organisms in 
relation to their coi cracting agencies). Tne quality of the 
implementation of activities is likewise evaluated- 
procedures of invitation to tender for the carrying out of 
training activities, the contractual process, criteria for 
identifying and choosing publics, etc. 

I Local Economic Conditions 

Every programme is implemented in a localised context, 
characterised by socio-economic structures and 
processes and the particular dynamics of the labour 
market. These contextual elements determine more or 
less intensely both the impact of the measures on their 
beneficiaries and the carrying out of training and entry 
activities on tiie local level (specific local characteristics 
of the population involved, local structure of training 
supply, needs of local businesses, etc.). 
At present, however, the extent to which these 
socio-economic data are taken into account in the 
anah^sis of programme effects remains limited. Only a 
few evaluation projects attempt to place their results in 



perspective with the characterisUcs of the local 
economy: growth rate and changing employment, rate 
and structure of unemployment, specific features of 
economic activities. Because it is very mobile the 
short-run economic situation has to be incorporated 
from the very conception of the entire observation and 
not simply injected as a last resort at the time of the 
statistical evaluations. 



* * 



The standardised quantitative tools (sample groups 
retrospective surveys) have attained a technical level 
that allows them maximum operation. At the same time 
the new, qualitative monographic analyses that take 
into account explanatory variables going beyond 
socio-demographic Indicators are beginning to 
proliferate in Europe. The next methodologic?! challenge 
is *n articulate these two levels and reduce the rupture 
DC n quantitative and qualitative, macro- and 
microeconomlc, longitudinal and synchronic, sti-uctural 
and biographical approaches. A greater formalisation of 
qualitative investigations is certainly a necessary first 
step, and CEREQ has made this its goal, tiarough 
among others, a study of the impact of the local 
environment on conditions of entry into working life 
among young people 

Article written by Annie Bouder, Jean-Paul Cadet 
Didier Demaziere (CLERSE-LASTREE)! 

CJaudine Roman! 
Crranslation by Miriam Rosen) 
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